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Abstract 

The constant emphasis on maintaining a high level of quality in the Open Distance Learning (ODL) self di¬ 
rected courses delivered at Wawasan Open University (WOU) demands the accurate assessment of course 
learning outcomes (CLO). The summative and formative assessment components of each course module 
are, in theory, designed to effectively measure the mastery of a set of CLO by an adult learner. However, in 
reality, there is no real scrutiny of the assessments at WOU to ascertain whether a learner has achieved the 
expected mastery of a CLO. The general assumption is that the assessment marks are an indication of the 
mastery of a particular set of CLO measured in a specific assessment component. With the aim of identifying 
whether assessment marks are indeed reflective of the mastery of a set of CLO, a study was conducted to 
implement the “Person eCollege Learning Outcome Manager” software application. This paper discusses the 
methodology and findings of this study and provides several suggestions for the smooth implementation of 
the Learning Outcome Manager. 

Keywords: Assessment; Course Learning Outcomes; Learning Outcome Manager; Open Distance Learning; 
Outcome Based Education; Pearson eCollege 


Introduction 

Being an Open Distance Learning (ODL) institution, Wawasan Open University (WOU) adopts a 
blended method for course delivery to its adult learners. At the undergraduate level, the learners 
undertake courses of five credit hours in self directed mode using purpose designed course mate¬ 
rials supplemented by brief face to face tutorial sessions and an online learning management 
system. Due to the lifelong learning aspect of the business where the focus on learning outcomes 
is considered crucial (Hussey & Smith, 2003), WOU invests heavily in the development and con¬ 
tinuous quality improvement of the self directed course materials which undergo a comprehensive 
instructional design process (Abeywardena, 2013). This, in turn, makes the materials suitable for 
learner centric Outcome Based Education (OBE) (McNeir, 1993). Each course is specifically designed 
to promote mastery of a particular set of course learning outcomes (CLO) by a learner. The CLO 
are further subdivided into unit learning outcomes (ULO), which allow learners to self assess their 
mastery using formative assessment components. The combination of CLO and ULO contribute 
holistically to the achievement of the program learning outcomes. Despite the insights provided by 
the formative assessment component into the learners’ mastery of CLO (Black & Wiliam, 1998) the 
lack of class room based activities in ODL makes the summative assessment component critical in 
assessing the mastery of CLO. 

The summative assessment at WOU is twofold: (i) continuous assessment in the form of Tutor 
Marked Assignments (TMA); and (ii) final proctored exam. However, this summative assessment 
component is not scrutinised in detail to assess the mastery of the CLO by the learners. In this 
regard, the general assumption being made is that the assessment marks are reflective of the 
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learners’ mastery of the CLO; which implies that the higher the marks the better the mastery of the 
CLO. Even though there have been studies conducted such as the one by Gijbels et al. (2005) 
which identifies a correlation between the mastery of CLO and the Grade Point Average (GPA), no 
such study has been conducted to identify the validity of this assumption in the context of the ODL 
environment at WOU. For the purpose of gaining an understanding of the correlation between the 
marks and the mastery of the CLO by WOU students, a study was conducted using the Pearson 
eCollege Learning Outcome Manager —which will be referred to as the Learning Outcome Manager 
for the remainder of this paper. 

The learning outcome manager is a web based software platform that allows academics to 
monitor the mastery of CLO by their students in both formative and summative assessments. The 
experiment was conducted as a retrospective study by superimposing the mastery of CLO on the 
assessment marks. The major contribution of this paper is the insight it provides into the correlation 
between summative assessment marks and the mastery of CLO by adult ODL learners. The rest 
of the paper is organised into four parts, which consist of methodology, results, discussion and 
conclusion. 

Methodology 

This particular study was conducted over a six-month period in the beginning of 2011. An independ¬ 
ent academy was created for WOU on the Learning Outcome Manager that allowed the detailed 
tracking of students’ progress for a particular ODL course. TCC123/05 Visual Programming , a five 
credit hour lower level technical course, equivalent to a first year course of a conventional university, 
specialising in Visual Basic.Net (VB.Net) was used as the test case for the study. This course was 
purposely chosen as the test case taking into account the total of 71 learners who had completed 
the course in the July semester of 2010. These learners were geographically distributed among 
four learning centers located across Malaysia. 

In theory, upon successful completion of the course the learners would have mastered five CLO 
as shown in Table 1. The summative assessment for this particular course comprised of (i) three 
TMA, which are a combination of theory and laboratory exercises contributing 50% to the final score; 
and (ii) one proctored examination contributing the remaining 50% to the final course score. The 
learners were required to obtain a minimum mark of 40% for both components to pass the course. 
In an attempt to ensure that the use of the Learning Outcome Manager would not interfere with the 
assessment and feedback provided to the students, the study was conducted retrospectively. As 
such, the Learning Outcome Manager was setup to superimpose the mastery of CLO on top of the 
assessment marks that had already been awarded through an independent exercise conducted 
prior to the study. 


Table 1: Course Learning Outcomes (CLO) for TCC123/05 Visual Programming 



Course Learning Outcome (CLO) 

1 . 

Discuss the principles of object oriented programming 

2. 

Write the basic elements of subroutines and functions in Visual Basic programs 

3. 

Create graphical user interface for Windows applications 

4. 

Develop web database applications 

5. 

Construct effective data structures and implement advanced object oriented programming approach 
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Four classes were created on the Learning Outcome Manager representing each of the four 
learning centers. Additionally, student accounts were created for each student enrolled in a particu¬ 
lar class. In a real-world scenario, the students would have been able to track their progress through 
the Learning Outcome Manager throughout the duration of their study. However, this feature was 
disabled for the purposes of the study as it was conducted retrospectively. Only the Student ID was 
used to identify the student in the system to ensure anonymity and unbiased evaluation. 

For the effective measurement of the mastery of CLO (i) each CLO was granulated into smaller 
ULO; (ii) a rubric comprising of a three point Likert scale was identified which measured the mastery 
of ULO in terms of needs improvement, meets the requirement or exceeds the requirement, and 
(iii) learning statements were identified which measured the mastery of a ULO against the rubric. 
Following this exercise, the rubric (Appendix A) was setup on the Learning Outcome Manager. 
Subsequently, the corresponding CLO in the rubric were mapped against the summative assess¬ 
ments as shown in Table 2. An assumption was made from the outset that the average of the 
mastery of each ULO would determine the overall mastery of the corresponding CLO. i.e. an aver¬ 
age mastery of 2/3 would be deemed as meeting the requirement with respect to the mastery of 
the CLO. 

Upon setup of the system, the academic in charge of the course did the data entry of the assess¬ 
ment marks. Following it, the TMA and answer scripts of the exam were re-evaluated to determine 
the mastery of the ULO by the learners. This re-evaluation was conducted against the rubric already 
setup on the Learning Outcome Manager. The numerical marks plus the letter grades were recorded 
in the Learning Outcome Manager for further analysis. Once the re-evaluation was complete, 
detailed reports were generated using the system in MS Excel format. The Pearson product-moment 
correlation coefficient (r) was used to determine the correlation between the assessment marks and 
the mastery of CLO. The statistical analysis was conducted using the MS Excel software applica¬ 
tion. 

Results 

The mastery of the CLO was mapped against the assessment marks for the exam and the three 
TMA in a scatter plot as shown in Figure 1. The Pearson product-moment correlation coefficient (r) 
between mastery of CLO and assessment marks was calculated for each assessment as shown in 
Table 3. 

Discussion 

According to the results of the statistical test (Table 3), there is a strong positive correlation (r « 1) 
between the mastery of the CLO and the assessment marks. This provides some indication that 


Table 2: Mapping of CLO to assessment 


Assessment 

Course Learning Outcome 

CLOI 

CL02 

CL03 

CL04 

CL05 

Proctored Exam 

V 

V 



V 

TMA1 


V 

V 



TMA2 

V 

V 




TMA3 



V 

V 

V 
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Figure 1: Mastery of CLOs Vs. Assessment Marks 


Table 3: Pearson product-moment correlation coefficient (r) between mastery of CLO and 

assessment marks 


Assessment 

Number of Learners 

(r) 

Exam 

48 

0.903748 

TMA1 

66 

0.768821343 

TMA2 

61 

0.843116626 

TMA3 

54 

0.816461787 


the marks of an assessment can be used to ascertain the mastery of the CLO. However, the cor¬ 
relation does not indicate which range of marks would be representative of meeting the requirement 
with respect to the mastery of CLO. To identify the possible correspondence between the assess¬ 
ment marks and the mastery of the CLO in relation to the rubric, the summative assessment marks 
were plotted against the mastery of the CLO in a scatter plot (Figure 1). By analysing the scatter 
plot, it was identified that the marks ranging from 60% to 80% are approximately indicative of the 
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Table 4: Correspondence between marks, grades and CLO 


Marks 

> 75 

60-74 

50-59 

46-49 

40-45 

< 39 

Grades 

A 

B 

C 

C- 

D 

F 

Mastery of CLO 

exceeds the 
requirement 

meets the requirement 

needs improvement 


learner’s mastery of the CLO. It is further approximated from Figure 1 that marks less than 40% 
are indicative of the need for improvement. This bears a resemblance to the passing mark of the 
summative assessments at WOU, which is set at 40%. Assessment marks beyond 80% are approx¬ 
imately indicative of the learner exceeding the requirement as far as the mastery of CLO is con¬ 
cerned. Referring to Table 4, it can be seen that the overall grade obtained by WOU learners in 
their summative assessment has an approximate correspondence to the level of mastery of CLO. 

Despite the fact that the statistical analysis is indicative of a correlation between the summative 
assessment marks and the mastery of CLO, it must be noted that the correspondence between the 
rubric and the assessment marks is only an approximation. This is mainly due to the breadth of the 
three point Likert scale used in the rubric to measure the mastery of the CLO. In the context of the 
Visual Programming course, given the technical nature of its content, mastery of certain ULO can 
only be measured as either meeting or not meeting the requirement. This skews the measurement 
to 1/2 instead of 2/3. In contrast, the achievement of certain ULO requires a scale larger than three 
points to identify the mastery in between meeting the requirement and exceeding the requirement. 
Therefore, it can be concluded that a more granular CLO assessment rubric is needed for effectively 
identifying a potential correspondence between the assessment marks and the mastery of CLO. 
This argument is further strengthened by the conclusions of the parallel study that used the three 
credit hour lower level foundation course Advanced Writing Skills for University Studies, which deals 
with content in the social science domain. In her paper detailing this study, Emmanuel (2011, p. 6) 
states 

. .when the three point Likert scale had to be created, it led the course coordinator (academic) to exam¬ 
ine in detail the criteria for each category to a great extent... to be able to show clear lines of delineation 

among the categories. This involves the ability to write good rubrics.” 

Emmanuel (2011, p. 6) further states: “.. .for course coordinators who come from industrial back¬ 
grounds, workshops need to be organised in this area otherwise the LOM (Learning Outcome 
Manager) efforts will be hampered.” 

As such, it can be noted that the accurate creation of the rubric for measuring the mastery of the 
CLO is a critical factor in the success of the whole Learning Outcome Manager implementation 
process. However, it must also be noted that while the increased granularity of the rubric contributes 
to the increased accuracy of the measurement, it also contributes to the fatigue of the assessor. 
Thus, the key is to find the correct balance between the two. 

Conclusion 

A study was initiated at Wawasan Open University (WOU) in collaboration with Pearson eCollege 
to identify a possible correlation between assessment marks and the mastery of course learning 
outcomes (CLO) in a self directed adult Open Distance Learning (ODL) environment. During the 
course of the study, it was realised that the implementation of the Learning Outcome Manager 
software application needs to be done holistically from inception to conclusion of an ODL course in 
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order to maximise the return on investment. As such, the implementation plan needs to take into 
consideration the design, development, delivery, assessment, feedback and revision stages of the 
course development lifecycle. 

The study further exposed the strengths and weaknesses of the present assessment strategies 
at WOU especially with respect to measuring the mastery of CLO. From the results, it can be seen 
that the summative assessments implicitly take this aspect into consideration. However, the need 
for an explicit attempt at measuring the mastery of the CLO needs to be seriously considered in 
the spirit of Outcome Based Education (OBE). Furthermore, the effectiveness of using a detailed 
rubric for measuring the mastery of CLO was also brought to the attention of the stakeholders. 
Another point that was brought to view was that certain CLO were being assessed multiple 
times in the summative assessment. In this regard, the study urged the academics to seriously 
re-evaluate their assessments taking into consideration the concepts in OBE. 

Overall, the Learning Outcome Manager was found to be a more comprehensive and effective 
method of assessing the mastery of CLO. The ability of the system to track the mastery of the 
smaller ULO in addition to the larger CLO provides all stakeholders a real-time view of the progress 
made by the students and the class as a whole. This, in turn, acts as a continuous quality improve¬ 
ment mechanism whereby the key stakeholders are provided with the ability to tailor learning 
activities and assessments to ensure the mastery of most CLO by the learners. Furthermore, the 
Learning Outcome Manager provides a usable yardstick for the accurate assessment of OBE in 
ODL environments which doubles as a valuable enabler especially in accreditation exercises. 
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Appendix A 

Rubric for measuring the mastery of Unit Learning Outcomes (ULO) for a particular Course Learn¬ 
ing Outcome (CLO). 


CLOI: Discuss the principles of object oriented programming 


Exceeds the 
Requirement 

Meets the Requirement 

Needs Improvement 

UL01.1 

Identify the differences 
between procedural and 
object-oriented program¬ 
ming. 


Being able to differentiate 
between procedural and 
object-oriented 
programming. 

Not being able to 
differentiate between 
procedural and 
object-oriented 
programming. 

UL01.2 

Describe the Handlers 
Design pattern. 


Being able to explain the 
role of events. 

Not being able to explain 
the role of events. 

UL01.3 

Recall the basic concepts 
of object-oriented 
technology. 

Being able 
to explain 
information hiding, 
inheritance and 
polymorphism. 

Being able to explain 
information hiding and 
inheritance. 

Not being able to explain 
information hiding and 
inheritance. 

UL01.4 

Explore the Visual Basic 
2008 Integrated 
Development 

Environment (IDE). 


Being able to create a 
project in the IDE. 

Not being able to create 
a project in the IDE. 

UL01.5 

Develop a simple GUI 
event-driven program 
using Visual Basic. 


Being able to create a 
button click event. 

Not being able to create 
a button click event. 

UL01.6 

Apply the Visual Basic 
program to create a 
simple application. 


Being able to create a 
simple program to 
display a word when a 
button click event 
happens. 

Not being able to create 
a simple program to 
display a word when a 
button click event 
happens. 


CL02: Write the basic elements of subroutines and functions in Visual Basic programs. 


Exceeds the 
Requirement 

Meets the 

Requirement 

Needs Improvement 

UL02.1 

Write a simple 
program using Visual 
Basic. 


Being able to create a 
console application to 
output a string. 

Not being able to create 
a console application to 
output a string. 

UL02.2 

Explain and apply primi¬ 
tive data types including 
integers, strings, and 
dates. 


Being able to use inte¬ 
ger, string and date in a 
console application. 

Not being able to use 
integer, string and date 
in a console application. 
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CL02: Write the basic elements of subroutines and functions in Visual Basic programs. 


Exceeds the 
Requirement 

Meets the 

Requirement 

Needs Improvement 

UL02.3 

Use arithmetic and 
logical operators to 
perform calculations. 


Being able to use AND, 
OR, NOT, +, -, *, /, 

MOD in a console 
application for 
calculations. 

Not being able to use 
AND, OR, NOT, +, -, *, 

/, MOD in a console 
application for 
calculations. 

UL02.4 

Apply code selection 
and repetition 
statement. 


Being able to use IF... 
Else, Select... Case, 

For, While loops for 
code selection and 
repetition in a console 
application. 

Not being able to use 

IF.. .Else, Select... 

Case, For, While loops 
for code selection and 
repetition in a console 
application. 

UL02.5 

Create and apply 
methods to a 
program. 

Being able to create 
a method, pass 
parameters to the 
method, call the method 
from within the main 
method and return 
values to the calling 
method. 

Being able to create a 
method and call the 
method from within the 
main method. 

Not being able to create 
a method and call the 
method from within the 
main method. 


CL03: Create graphical user interface for Windows applications 


Exceeds the 
Requirement 

Meets the 

Requirement 

Needs Improvement 

UL03.1 

Design and build a 
graphical Windows 
application using GUI 
controls. 

Being able to add GUI 
controls to a form 
following industry 
standards for GUI 
development. 

Being able to add GUI 
controls to a form. 

Not being able to add 

GUI controls to a form. 

UL03.2 

Create and manipulate 
GUI controls. 

Being able to modify the 
properties of a control 
to suit the requirement 
while adhering to 
industry standards for 

GUI controls. 

Being able to modify the 
properties of a control 
to suit the requirement. 

Not being able to 
modify the properties of 
a control to suit the 
requirement. 

UL03.3 

Use the Windows 
Presentation Foundation 
(WPF) to draw simple 
shapes. 

Being able to create a 
WPF project and draw 
complex geometrical 
shapes according to 
the specifications. 

Being able to create a 
WPF project and draw a 
simple geometrical 
shape according to the 
specifications. 

Not being able to create 
a WPF project and draw 
a simple geometrical 
shape according to the 
specifications. 
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UL03.4 

Apply simple SQL 
queries. 

Being able to 
manipulate data in a 
database using the 
SELECT, INSERT, 
UPDATE, DELETE 
queries. 

Being able to get data 
from a database using 
the SELECT query. 

Not being able to get 
data from a database 
using the SELECT 
query. 

UL03.5 

Generate database 
connections and create 
LINQ to SQL objects. 

Being able to create a 
database connection, 
create a LINQ to SQL 
object and use the 

LINQ to SQL object in 
an application. 

Being able to create a 
database connection 
and create a LINQ to 

SQL object. 

Not being able to create 
a database connection 
and create a LINQ to 

SQL object. 


CLQ4: Develop web database applications 


Exceeds the 
Requirement 

Meets the 

Requirement 

Needs Improvement 

UL04.1 

Develop a Web 
application using 
ASP.NET. 

Being able to start an 
ASP.Net project and 
create multiple pages. 

Being able to start an 
ASP.Net project and 
create a default page. 

Being able to start and 
ASP.Net project but not 
being able to create a 
default page. 

UL04.2 

Create Web Forms. 

Being able to insert a 
from into a page with 
form controls for 
gathering information 
and adding the code 
behind to submit the 
information to the 

server. 

Being able to insert a 
from into a page with 
form controls for 
gathering information. 

Not being able to insert 
a from into a page with 
form controls for 
gathering information. 

UL04.3 

Implement data 
validation controls. 

Being able to bind the 
data validation controls 
to the form and output 
informative information 
to the user. 

Being able to bind the 
data validation controls 
to the form. 

Not being able to bind 
the data validation 
controls to the form. 

UL04.4 

Use cookies to obtain 
information about users. 

Being able to create, 
deploy, update, retrieve 
and delete cookies 
and use the cookie 
information in the 
application. 

Being able to create, 
deploy, update, retrieve 
and delete cookies. 

Not being able to 
create, deploy, update, 
retrieve and delete 
cookies. 

UL04.5 

Connect to a database 
in ASP.NET. 

Being able to add a 

LINQ to SQL object into 
an ASP.Net application 
and use the data from 
the database in the 
application. 

Being able to add a 

LINQ to SQL object into 
an ASP.Net application. 

Not being able to add a 
LINQ to SQL object into 
an ASP.Net application. 
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CL05: Construct effective data structures and implement advanced object oriented 

programming approach. 


Exceeds the 
Requirement 

Meets the 

Requirement 

Needs Improvement 

UL05.1 

Create object-oriented 
programs. 

Being able to create 
classes, objects and 
use objects to interact 
with the class in an 
application. 

Being able to create 
classes and objects. 

Not being able to create 
classes and objects. 

UL05.2 

Demonstrate data 
structure programming 
methods in Visual 

Basic. 

Being able to program 
the logic behind stack, 
queue and list using 
arrays. 

Being able to implement 
stack, queue, list and 
arrays in applications. 

Not being able to 
implement stack, queue, 
list and arrays in 
applications. 

UL05.3 

Use generic and 
collection methods as a 
new tool in Visual 

Basic. 

Being able to use 
generic methods and 
collections to complete 
tasks in applications. 

Being able to implement 
generic methods and 
collections in 
applications. 

Not being able to 
implement generic 
methods and collections 
in applications. 

UL05.4 

Create and manipulate 
static and dynamic data 
structures. 

Being able to create 
and use arrays, stacks, 
queues and linked lists 
to complete tasks in 
applications. 

Being able to create 
and use arrays, stacks, 
queues and linked lists. 

Not being able to create 
and use arrays, stacks, 
queues and linked lists. 
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